Summarise the Monte-Carlo test_convergence ensembles by default (breaking) - #4056
Summarise the Monte-Carlo test_convergence ensembles by default (breaking)#4056ChrisRackauckas-Claude wants to merge 2 commits into
Conversation
test_convergence retained the full EnsembleSolution for every step size in _solutions and only reduced them to error summaries after the loop, so a Monte-Carlo study's peak was the sum over all step sizes rather than the largest one. The weak-convergence groups run 5e5-1e6 trajectories over 5-8 step sizes, which is what put OOPWeakConvergence and IIPWeakConvergence past a 16 GB runner. Add `retain_solutions` to test_convergence. When false, each step size's ensemble is reduced by calculate_ensemble_errors as soon as it is solved and then stripped to a single representative trajectory, so only one ensemble is alive at a time. The error statistics are computed from the full ensemble either way -- only `sim.solutions[i].u` is shortened -- and the flag defaults to true, so existing callers are unaffected. Measured on the two groups, single run each, JULIA_NUM_THREADS=8: iip_weak peak 25.10 -> 7.93 GiB (3.2x), wall 7:29 -> 6:35 oop_weak peak 22.93 -> 7.30 GiB (3.1x), wall 7:33 -> 7:00 Both now fit the standard self-hosted pool, so drop the high-memory runner label from OOPWeakConvergence and IIPWeakConvergence. That label is carried by no runner in the fleet, so those groups have been sitting queued for 24 h and expiring rather than running at all (see SciML#4030). That the statistics are untouched is checked directly rather than argued: a 20000-trajectory SROCK2 study run both ways gives bitwise identical `𝒪est` and `errors` while the retained object shrinks 489x. test/retain_solutions_tests.jl covers that, the trajectory count actually dropping, and the expected_value path being left alone. Not applied to SROCKC2WeakConvergence: measured with the flag it still peaks at 34.15 GiB, because its cost is a single 3e6-trajectory ensemble rather than retention across step sizes. Nothing here helps that, so its label stays. Note separately that the group fails its assertions when it does run -- 𝒪est[:weak_final] comes out 1.567 and 1.563 against a `< 0.4` band around 2 -- which has gone unseen because the group has never been scheduled. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013WW2jGeoWZVZu7AiscUNSz
BREAKING: DiffEqDevTools 3.1.5 -> 4.0.0. test_convergence kept the full EnsembleSolution for every step size in _solutions and only reduced them to error summaries after the loop, so a Monte-Carlo study's peak was the sum over all step sizes rather than the largest one. At 5e5-1e6 trajectories over 5-8 step sizes that is tens of GiB of solutions retained only to compute a mean, and it is what put OOPWeakConvergence and IIPWeakConvergence past a 16 GB runner. Reduce each step size's ensemble as soon as it is solved and strip it to one representative trajectory, so only one ensemble is alive at a time. This is now the default; `retain_solutions = true` restores the old behaviour for callers that want the paths themselves. Measured on the two groups, JULIA_NUM_THREADS=8: iip_weak peak 25.10 -> 7.93 GiB (3.2x), wall 7:29 -> 6:35 oop_weak peak 22.93 -> 7.30 GiB (3.1x), wall 7:33 -> 7:00 Both now fit the standard self-hosted pool, so drop the high-memory runner label from those two groups. No runner in the fleet carries that label, so they have been queueing for 24 h and expiring rather than running (SciML#4030). Why this is breaking: `ConvergenceSimulation.solutions` is documented, and the docs said it "Holds all the PdeSolutions" — which is exactly the promise being changed. `Base.getindex(sim, i, I...)` forwards into it, so `sim[i][j]` for j > 1 now errors on the Monte-Carlo method. The field documentation is updated to match. Only the Monte-Carlo method changes; the ODE/DAE methods of test_convergence do not take the keyword and are untouched. The one place in this repo that indexes ensemble trajectories, PL1WM.jl:234 and :274, goes through the expected_value path, which builds no EnsembleTestSolution and so never reduces. That the statistics are unchanged is checked rather than argued: a 20000-trajectory SROCK2 study run both ways gives bitwise identical `𝒪est` and `errors` while the retained object shrinks 489x. test/retain_solutions_tests.jl covers that, pins the new default, checks the trajectory count actually drops, and checks the expected_value path is left alone — 39 assertions. Not applied to SROCKC2WeakConvergence: measured, it still peaks at 34.15 GiB, because its cost is a single 3e6-trajectory ensemble rather than retention across step sizes. Its label stays. Separately, that group fails its assertions when it runs — 𝒪est[:weak_final] of 1.567 and 1.563 against a < 0.4 band around 2 — which has gone unseen because it has never been scheduled. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013WW2jGeoWZVZu7AiscUNSz
|
Heads-up: I opened #4060 for the same problem before seeing this. Two sessions converged on #4037 independently. They should not both merge — flagging the differences so one can be picked rather than reconciled later. Where the reduction happens. This PR reduces each ensemble after That difference is exactly what this PR identifies as its own limitation:
#4060 is that
so the Breaking vs not. This PR changes the default and moves DiffEqDevTools to 4.0.0, touching the compat pin in 45 One measurement to double-check. This PR reports I did hit the same |
|
Closing in favour of #4060, which solves the same issue by reducing each trajectory during the solve through the ensemble's
#4060 takes Two things from here are being carried over rather than lost:
Also worth not losing: this PR's observation that Thanks — the analysis here was sound and the parts of it that were better than mine are in #4060. |
…uring `test_convergence` kept one full solution object per trajectory per step size. A `RODESolution` carries the solver cache, the noise process, the problem and its interpolation, none of which a convergence study needs — it wants the per-trajectory errors and the endpoints the weak error is formed from. At the trajectory counts the weak-convergence tests use, that gap is the difference between passing and being OOM-killed: `SROCKC2WeakConvergence`, `IIPWeakConvergence` and `OOPWeakConvergence` all died at 15.6 GiB in a 16 GiB cgroup with no test output, which is the "runner lost communication with the server" signature reported in SciML#4036. Trajectories are no longer retained by default. Three things compose: - The `EnsembleProblem` is built with an `output_func` that reduces each trajectory to a `ConvergenceTrajectory` as it is solved, so the full solutions never coexist and the peak is bounded, not just the retention. `ConvergenceTrajectory` stores one-tuples and a `NamedTuple` rather than one-element `Vector`s and the solver's error `Dict`; indexing and key lookup are unchanged, but the `Dict` alone dominated everything else retained once a study runs to millions of trajectories. - `calculate_ensemble_errors` moves inside the solve loop, so each step size's ensemble is summarised and released before the next is solved rather than all of them being held at once. - `_drop_trajectories` strips the summarised ensemble to one representative, since the reduced objects are dead weight once the errors are computed. `retain_solutions = true` restores the old behaviour for callers that want the paths themselves. The reduction is skipped, and the solutions retained, where it cannot apply: a caller-supplied `EnsembleProblem` owns its own `output_func`, `expected_value` forms the weak error from the trajectory values directly, and `weak_timeseries_errors`/`weak_dense_errors` need the whole timeseries. Drop `uEltype` from the `ConvergenceSimulation` constructor — it was assigned and never used, and was the only thing requiring `solutions[i].u[j]` to be a full solution. Measured on the weak files, 16 GiB cgroup, JULIA_NUM_THREADS=2: weak_srockc2.jl OOM 15.61 GiB @3m29s -> exit 0, 23m41s, 2.81 GiB oop_weak.jl OOM 15.62 GiB @23m50s -> exit 0, 13m08s, 1.32 GiB additive_weak.jl (same group) -> exit 0, 3m22s, 1.04 GiB Order estimates are bit-identical with and without the reduction. With no group needing more than 2.81 GiB, remove the `high-memory` runner pin from all eight groups that carried it. The five StochasticDiffEqWeak groups needed no change — they already reduce through their own `output_func` and peak at 0.71-2.09 GiB, so the label was simply wrong for them. BREAKING: DiffEqDevTools 3.1.5 -> 4.0.0. `ConvergenceSimulation.solutions` no longer holds every trajectory for the Monte-Carlo method, and `sim[i, j]` forwards into it, so `j > 1` now reaches past the representative. The field documentation is updated and the compat pin moves "3" -> "4" in 40 Project.toml files. The ODE/DAE methods do not take the keyword and are unchanged. The reduce-by-default design and `drop_trajectories` are taken from SciML#4056, which is closed in favour of this. Fixes SciML#4037 Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UwLXp5WY1uiqPun7qhwxeU
…uring `test_convergence` kept one full solution object per trajectory per step size. A `RODESolution` carries the solver cache, the noise process, the problem and its interpolation, none of which a convergence study needs — it wants the per-trajectory errors and the endpoints the weak error is formed from. At the trajectory counts the weak-convergence tests use, that gap is the difference between passing and being OOM-killed: `SROCKC2WeakConvergence`, `IIPWeakConvergence` and `OOPWeakConvergence` all died at 15.6 GiB in a 16 GiB cgroup with no test output, which is the "runner lost communication with the server" signature reported in SciML#4036. Trajectories are no longer retained by default. Three things compose: - The `EnsembleProblem` is built with an `output_func` that reduces each trajectory to a `ConvergenceTrajectory` as it is solved, so the full solutions never coexist and the peak is bounded, not just the retention. `ConvergenceTrajectory` stores one-tuples and a `NamedTuple` rather than one-element `Vector`s and the solver's error `Dict`; indexing and key lookup are unchanged, but the `Dict` alone dominated everything else retained once a study runs to millions of trajectories. - `calculate_ensemble_errors` moves inside the solve loop, so each step size's ensemble is summarised and released before the next is solved rather than all of them being held at once. - `_drop_trajectories` strips the summarised ensemble to one representative, since the reduced objects are dead weight once the errors are computed. `retain_solutions = true` restores the old behaviour. The reduction is skipped, and the solutions retained, where it cannot apply: a caller-supplied `EnsembleProblem` owns its own `output_func`, `expected_value` forms the weak error from the trajectory values directly, and `weak_timeseries_errors` and `weak_dense_errors` need the whole timeseries. Drop `uEltype` from the `ConvergenceSimulation` constructor — it was assigned and never used, and was the only thing requiring `solutions[i].u[j]` to be a full solution. Measured on the weak files, 16 GiB cgroup, JULIA_NUM_THREADS=2: weak_srockc2.jl OOM 15.61 GiB @3m29s -> exit 0, 23m41s, 2.81 GiB oop_weak.jl OOM 15.62 GiB @23m50s -> exit 0, 13m08s, 1.32 GiB additive_weak.jl (same group) -> exit 0, 3m22s, 1.04 GiB Order estimates are bit-identical with and without the reduction. With no group needing more than 2.81 GiB, remove the `high-memory` runner pin from all eight groups that carried it. The five StochasticDiffEqWeak groups needed no change — they already reduce through their own `output_func` and peak at 0.71-2.09 GiB, so the label was simply wrong for them. Shipped as a minor bump, 3.1.5 -> 3.2.0. A search of public Julia code found no consumer of the trajectories this stops retaining: `sim.solutions` appears four times in total, two of which are this package's own source and one an archived pre-split mirror, and the only real use is PL1WM.jl in this repo. That one now passes `retain_solutions = true` explicitly at the four call sites whose sims are compared trajectory-by-trajectory, rather than relying on the caller-supplied-`EnsembleProblem` skip. The two downstream SDE callers (DiffEqGPU.jl, SciMLBenchmarks) pass their own `EnsembleProblem` and are unaffected, and `analyticless_test_convergence` is untouched. The reduce-by-default design and `_drop_trajectories` are taken from SciML#4056, which is closed in favour of this. Fixes SciML#4037 Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UwLXp5WY1uiqPun7qhwxeU
…uring (#4060) `test_convergence` kept one full solution object per trajectory per step size. A `RODESolution` carries the solver cache, the noise process, the problem and its interpolation, none of which a convergence study needs — it wants the per-trajectory errors and the endpoints the weak error is formed from. At the trajectory counts the weak-convergence tests use, that gap is the difference between passing and being OOM-killed: `SROCKC2WeakConvergence`, `IIPWeakConvergence` and `OOPWeakConvergence` all died at 15.6 GiB in a 16 GiB cgroup with no test output, which is the "runner lost communication with the server" signature reported in #4036. Trajectories are no longer retained by default. Three things compose: - The `EnsembleProblem` is built with an `output_func` that reduces each trajectory to a `ConvergenceTrajectory` as it is solved, so the full solutions never coexist and the peak is bounded, not just the retention. `ConvergenceTrajectory` stores one-tuples and a `NamedTuple` rather than one-element `Vector`s and the solver's error `Dict`; indexing and key lookup are unchanged, but the `Dict` alone dominated everything else retained once a study runs to millions of trajectories. - `calculate_ensemble_errors` moves inside the solve loop, so each step size's ensemble is summarised and released before the next is solved rather than all of them being held at once. - `_drop_trajectories` strips the summarised ensemble to one representative, since the reduced objects are dead weight once the errors are computed. `retain_solutions = true` restores the old behaviour. The reduction is skipped, and the solutions retained, where it cannot apply: a caller-supplied `EnsembleProblem` owns its own `output_func`, `expected_value` forms the weak error from the trajectory values directly, and `weak_timeseries_errors` and `weak_dense_errors` need the whole timeseries. Drop `uEltype` from the `ConvergenceSimulation` constructor — it was assigned and never used, and was the only thing requiring `solutions[i].u[j]` to be a full solution. Measured on the weak files, 16 GiB cgroup, JULIA_NUM_THREADS=2: weak_srockc2.jl OOM 15.61 GiB @3m29s -> exit 0, 23m41s, 2.81 GiB oop_weak.jl OOM 15.62 GiB @23m50s -> exit 0, 13m08s, 1.32 GiB additive_weak.jl (same group) -> exit 0, 3m22s, 1.04 GiB Order estimates are bit-identical with and without the reduction. With no group needing more than 2.81 GiB, remove the `high-memory` runner pin from all eight groups that carried it. The five StochasticDiffEqWeak groups needed no change — they already reduce through their own `output_func` and peak at 0.71-2.09 GiB, so the label was simply wrong for them. Shipped as a minor bump, 3.1.5 -> 3.2.0. A search of public Julia code found no consumer of the trajectories this stops retaining: `sim.solutions` appears four times in total, two of which are this package's own source and one an archived pre-split mirror, and the only real use is PL1WM.jl in this repo. That one now passes `retain_solutions = true` explicitly at the four call sites whose sims are compared trajectory-by-trajectory, rather than relying on the caller-supplied-`EnsembleProblem` skip. The two downstream SDE callers (DiffEqGPU.jl, SciMLBenchmarks) pass their own `EnsembleProblem` and are unaffected, and `analyticless_test_convergence` is untouched. The reduce-by-default design and `_drop_trajectories` are taken from #4056, which is closed in favour of this. Fixes #4037 Claude-Session: https://claude.ai/code/session_01UwLXp5WY1uiqPun7qhwxeU Co-authored-by: ChrisRackauckas-Claude <accounts@chrisrackauckas.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
BREAKING: DiffEqDevTools 3.1.5 → 4.0.0.
The problem
test_convergencekept the fullEnsembleSolutionfor every step size in_solutions, reducing them to error summaries only after the loop:So the peak is the sum over step sizes rather than the largest. At 5e5–1e6 trajectories over 5–8 step sizes that is tens of GiB of solutions retained only to compute a mean, and it is what put
OOPWeakConvergenceandIIPWeakConvergencepast a 16 GB runner.The change
Each step size's ensemble is reduced as soon as it is solved and stripped to one representative trajectory, so only one is alive at a time. This is now the default.
retain_solutions = truerestores the old behaviour for callers that want the paths themselves.Measured
JULIA_NUM_THREADS=8, before/after on non-overlapping checkouts:iip_weakoop_weakBoth now fit the standard self-hosted pool, so the
high-memorylabel comes off those two groups — no runner in the fleet carries it, so they have been queueing for 24 h and expiring rather than running (#4030).Why this is breaking
ConvergenceSimulation.solutionsis documented, and the docs said:which is exactly the promise being changed.
Base.getindex(sim, i, I...) = sim.solutions[i][I]forwards into it, sosim[i][j]forj > 1now errors on the Monte-Carlo method. The field documentation is updated to match, and the compat pin moves"3"→"4"in 45Project.tomlfiles.Scope of the break, audited rather than assumed:
test_convergencedo not take this keyword and are untouched — including thesim[i].u[2]usage insymplectic_convergence.jl, which is aDynamicalODEProblem.PL1WM.jl:234and:274, comparing two algorithms path by path. It goes through theexpected_valuepath, which takes a reducingoutput_func, builds noEnsembleTestSolution, and so never reduces. Unaffected.That the statistics are unchanged is checked, not argued
A 20000-trajectory SROCK2 study run both ways:
test/retain_solutions_tests.jlcovers statistics preservation across𝒪est,errors,weak_errors,error_meansand the per-trajectoryerrorsvectors; pins the new default; checks the trajectory count actually drops; and checks theexpected_valuepath is left alone. 39 assertions, all passing.What this does not fix
SROCKC2WeakConvergenceis not helped. Measured with the reduction in place it still peaks at 34.15 GiB, because its cost is a single 3e6-trajectory ensemble rather than retention across step sizes. Itshigh-memorylabel stays. It needs either a lower trajectory count or the reducing-output_funcpattern the StochasticDiffEqWeak studies already use (8 bytes/trajectory instead of KiB).That group also fails its assertions.
𝒪est[:weak_final]came out 1.567 and 1.563 againstabs(𝒪 - 2) < 0.4, i.e. just under the 1.6 floor, at both 3e6 and 1e6 trajectories. This has gone unseen because the group has never once been scheduled in the CI history I looked at. Fixing its memory would turn "never runs" into "runs and fails", so it wants its own investigation into whether the tolerance or the method is at fault.Caveat on how this was measured
Pkg.test()forlib/StochasticDiffEqdoes not resolve on current master —ImplicitDiscreteSolve2.1.5 requiresNonlinearSolveBase≥ 2.40.0 while the registry tops out at 2.38.0 — so the two weak files were run directly in a purpose-built environment rather than through the group harness. Same shape as theNonlinearSolve4.24.0 floor that was breaking the GPU jobs, and it is why thelib/StochasticDiffEqandlib/DiffEqDevToolsgroups are currently red on master.Links
🤖 Generated with Claude Code
https://claude.ai/code/session_013WW2jGeoWZVZu7AiscUNSz